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FOREWORD 



In any list of axioms supporting the organized effort of formal schooling, one is 
paramount: without knowledge of results, there cannot be improvement . Seen from this 
light, the Educational Program Audit is a basic tool for educational improvement. 

The EPA insures objective feedback. If it is conducted in the spirit of redesign for 
quality assurance rather than inspection it will enlist the entliusiastic support of each 
professional worker because it will enhance both credibility and capabihty. 

Cohen, Turner, and Wiener have distilled the essence of the EPA in a straightforward, 
easy-to-follow format. From my personal knowledge of their dedication and expertise, as 
well as the soundness of the material represented in this handbook, I am convinced that this 
little pubhcation will make a major demonstrable contribution to both educational reform 
and renewal. 

As science is utterly dependent upon objective feedback and review, so must we in 
education discipline ourselves to this process. To do less is to forfeit the opportunity for a 
true profession. To do more is to lead the profession to new heights of service. 

Leon Lessinger 

Former US Associate Commissioner 
of Education and Callaway Professor 
of Education at Georgia State 
University 
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INTRODUCTION 



Only a few gcMicrations ago it was the custom in school districts for the fiscal records 
to be kept or not kept in whatever fashion the districts saw lit. In time, state legislatures 
required that district fiscal records be kept uniformly according to some rules of the game; 
the rules for keeping the books came to be written in the state*s accounting manual. 
About a generation ago, many legislatures required that fiscal records be audited by an 
agent independent of the district. There was much resistance at the time in the vein of, 
"It*s a duplication of effort: It*s a waste of money: Don*t you trust me?" Today, the 
school administrator points to his fiscal audit as unbiased verification that his records 
are in order. Such is usually not the case with his educational programs. 

It has too often been our custom in Education to initiate a new instructional program 
by assigning responsibility for its implementation to the most convenient administrator. 
The administrator may also be advised, sometimes almost as an afterthought, "Oh >es, 
and let me know lunv it turns out." After a school year of struggling to implement the 
new program in addition to his other duties, the administrator may be called upon to 
report to the Superintendent or the Board on "how it turned out." The Program 
Administrator, having a human amount of human frailties, and thus not wishing to appear 
inadequate before his administrative superiors may report that the parents liked it, the 
kids liked it. and it turned out not too badly. Small wonder then that reports of educators 
to their constituencies have less credibility that we would wish. When we assign 
responsibility for program operation and program evaluation to the same person, we have 
made the pitcher the umpire. 

In recent years, ihc requirement of an evaluation component in federally funded 
educational programs has been perceived as a means of reestablishing our credibility with 
(Mir various constituents. However, in locally funded programs, this step has often been 
egrogiously neglected or omitted. Even when this step is taken, it does not go far enough. 
Internal evaluation is analogous to asking the company bookkeeper, "How much money 
did we have left Jt the end of the year?" and then taking his word for it. The prudent 
businessman would be well advised to ask an independent auditing firm to verify the 
conclusion of the bookkeeper. Similarly, evaluating educational programs is better business 
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than not evaluating them. However, this too is an incomplete cycle. Our publics would 
be better served if we included the next step - verification of the evaluation by an 
educational program auditor. 

The EPA enters the educational cycle after the objectives have been determined and 
before the programs and criterion measures have been adopted, as indicated on the flow 
chart below. 




The sequential nature of the flow chart makes it appear that Audit follows Evaluation. 
In reality, the audit process begins with the hiring of the Auditor and continues through 
the life of the project. Tlus handbook is designed to assist the Auditor in this task. 
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CHAPTER I 
Federal Audit Requirements 



Independent accomplishment audits are required under a variety of federal titles; e.g., 
Title VII, Title' VIII, and portions of Title III, ESEA. The Manual for Project Applicants and 
Grantees for Title VII, ESEA , specifies that "all projects must provide for an independent 
educational accomplishment audit of the project to apprise school officials of the validity of 
their own evaluative process and data."' It further specifies "the final audit report will be 
prepared and submitted to the appropriate LEA personnel (school board, superintendent, 
project director, project evaluator) according to the contract time schedule. It will be the 
responsibility of the LEA to forward five copies of each final audit report to the United 
States Office of Education within thirty days of its receipt by the LEA. In addition, the 
number of copies required by the state educational agency should be submitted to that 
agency (unless it is the auditing agency)." 

There are indications that audit requirements may soon extend to Title I and 
Vocational Education Projects, and that before long, most federally funded programs may 
require an educational program audit. 

Before continuing with descriptions of auditors and educational program audits a brief 
comment is in order. Evaluation and audit are not the same. Evaluation may l^e conducted 
by personnel employed within a project or district, or it may be conducted by an outside 
contractor. The education program auditor is an outside, independent source whose purpose 
is to verify the reported results of the evaluation of the educational program and to assess 
the appropriateness of the evaluation teclmiques. 

"The independent educational audit is the device through which the public can hold its 
schools accountable, and also through which the school can learn how to improve its 
programs in order to meet the demands rightly made by its constituency."^ 



^The Manual for Project Applicants and Grantees for Programs under Bilingual Education Act, Title 
VII, ESEA, p. 11. 

^Lessinger, Leon, Every Kid a Winner , Simon and Schuster, New York; 1970, p. 79. 

O 
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Roles of the Educational Program Auditor 



The Educational Program Auditor brings a new professional role to education. He 
serves as a third party, free of local ties and interests, who verifies the results of the internal 
evaluation. = 

The Educational Program Auditor is : 

1. Independent from the program to be audited 

2. A reviewer and not a decision-maker 

3. A reporter and an observer 

4. A professional who exercises care and integrity in performing the audit 
examination and in preparing audit reports 

5. Equipped to innovate within his own profession and to encourage reforms in the 
schools 

The Tasks of the Educational Program Auditor Include : 

1. Critiquing the evaluation design 

2. Making alternative instrumentation and design suggestions 

3. Making alternative suggestions for data collection and analysis 

4. Assessing the extent to which the instructional materials and facilities are being 
used or prepared in accordance with the specifications set forth in the program 
proposal 

5. Conducting interviews, reviewing materials, and visiting sites 

6. Providing feed-back information to program administrators to help in improving 
program performance 

7. Making agreements with the users of the audit report on standards of evaluation 
and summarization 

8. Presenting information in such a manner that the information can be used by 
district, personnel in formulating judgments and making rational decisions 

9. Verifying the results reported in ttie evaluation report 
10. Comparing proposed practice with actual practice 
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Qualifications of an Educational Program Auditor : 

1. Generally the Program Auditor selected should have at least the same level of 
qualifications and skills in evaluation as the Program Evaluator. 

2. The Auditor should be independent from the local educational agency and should 
not have any connection ^ith the program that could inject bias into the audit 
function, 

3. The Auditor should be close enough to the project site so that cost would not 
preclude on-site visits to examine data, schedules, procedures, implementation, 
and findings. 

4. The Auditor should be able to provide a record of experience in program audits 
as well as theoretical capability. 

5. For projects federally funded under ESEA Titles VII and VIII, the Auditor must 
be approved by the United States Office of Education. 
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CHAPTER II 
The Pre Audit 



Review Documents 

Once the Auditor has been selected, he should be supplied with the preliminary 
program proposal, the proposed evaluation design, and any other information which the 
Program Director and the Evaluator believe to be relevant. In the case of federally funded 
projects this would usually consist of the guidelines governing the particular statute, the 
project proposal prepared by the district, and the grant document authorizing the 
expenditure of funds. Documents for locally funded programs would normally consist only 
of the program proposal and the local district authorization to expend funds. The Auditor 
should examine the documents for evidence of district commitment to the program in 
the form of official board and/or adn-inistrative action in the form of board minutes, 
policy, or regulation. 

Review Needs A^. ^ smeU 

The Auditor should examine the documents supplied him by the LEA to determine 
whether a needs assessment has been conducted and whether the proposed program is 
addressed to an identified need. He should examine whether alternatives for meeting this 
identified need were explored and whether a rationale waa developed for making choices 
among the alternatives. If not, it may be appropriate for the Auditor to meet with the 
Program Director and Evaluator at this point to discuss alternatives. Suggestions made 
at this meeting by the Auditor should be in writing and should be included in the final 
audit report. 

Review Management System 

The Auditor should examine whether management has provided for: 
Responsibilities Performance Indicators 

a. Logistical support. Allocation of manpower, money, materials. The 

program application is probably the best source 
for this information. 
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Rcsponsibilitics 
Monitoring the program. 



Performance Indicators 

Lines of authority and responsibility for the project 
should be clearly stated. The project proposal may 
contain this information; board minutes or district 
policy statements may be a source for this 
information. If the Auditor cannot discern from the 
documents presented to him what tlie Uncs of 
authority and responsibility are. ne should include 
this item in his discussion with the Prnicc% Director 
and Evaluator. His suggestions should be in writing 
and should be included in the final report. 



c. A training system for 
initial project 
implementation as well as 
a back-up training system 
if the project is to be 
continued or expanded. 
This information should 
be included in the project 
proposal. 



Interviews with project participants con reveal 
whether the goals and objectives of the program are 
understood and whether the participants arc willing 
and competent to execute the methods-means 
selected for the program. 
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d A communications 
system. 



The Auditor should examine whether provision has 
been made for information flow from project 
participants to management; between project 
connected participants; between the board and the 
general public. Information flow from project 
management to district management would normally 
be found in the monitoring system in b. above. The 
Auditor should be prepared to suggest alternatives if 
no information system is provided for in the 
documents. 
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R evicw Project Objectives 

It is important that the project objectives be presented in a straightforward fashion. 
The major focus of any proposal must be a specification of what the proposed project is 
intended to accomplish. It is essential that the EPA review the project objectives to assure 
sufficient technical quality as well as to make sure that the objectives of the project are, in 
fact, focused on the problem of interest. It is the function of the project evaluation team to 
work with the project director, curriculum specialists, and teachers in defining project 
objectives so that they can be measured and reflect clearly specified needs assessment. 
Operationally, this means that projec objectives must communicate to the interested 
observer v/hat tiie project participants should be ?;^le to accomplish at the conclusion of 
their exposure to the treatment involved in that project, the conditions under which they 
are to exhibit the desired behavior, and the overt behavior to be accepted as evidence of 
their having accomplished the objectives of the educational program. 

It is the function of the EPA to verify that project objectives are indeed stated in such 
a manner that they can be evaluated. The heart of an objective is specificity. In order to be a 
well-written objective, most experts agree that four components must be included: 

1. (WHO) A specific statement of the INDIVIDUAL(S) who will exhibit the 
behavior. 

2. (WHAT) The specific BEHAVIOR exhibited when ac omplished. 

3. (WHEN) The specific POP^T IN TIM^. the behavior is to be accomplished. 

4. (HOW WELL) The specif c CRITERIA OF SUCCESS to be obtained. 

An example of a measurable objewtive is as follows: Upon completion of the twelfth 
grade, every st ;icnt will demonstrate ability to meet basic minimum reading requirements 
of the present adult society by ^orrectly interpreting printed instructions and providing 
responses with 80% accuracy as verified by the instructor when given the following state or 
federal forms: 
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1. A sample state driver's license test. 
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2. An application for a social security card. 

3. An income tax return (simple form). 

When writing or validating the objectives of an educational program, the following 
should be adhered to: 
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Criteria 



Application 



Importance and Comprehensiveness. The stated 
objectives are important outcomes to s. 1: in 
solving the identified problem. 



"meet basic minimum reading 
requirements in present adult 
society." 



2. Measurability . Valid ways of determining how well 
each objective is achieved are stated separately or 
are implied in the objective. 



'as verified by the instructor." 



Ou tcomes Specified . The objectives describe 
outcomes, not actions or solutions expected to 
produce the outcomes. 



"correctly interpreting 
instructions and providing 
accurate responses with 80% 
accuracy." 



Conditions of Achievement. The conditions under 
which achievement of the objective can be 
demonstrated as stated. 



"upon completion of the 
twelfth graae every 
student . . . when given 
printed state or federal forms." 



5. Relevance and Precision . The objectives are stated 
as narrowly and precisely as possible without 
sacrificing importance. Thus, planners are not 
misled into planning irrelevant or marginal 
activities by vagueness of the statement. 



see statement of objective. 



Criteria 



Application 



6. Relationship to Action . Causal events by which see statement of objective, 
any plan of action might achieve the objective can 

be specified convincingly, 

7. Communication. Those persons who interpret and see statement of objective, 
use the objectives understand their intent. 

Critique Evaluation Design 

Many districts do not employ the specialized services of an evaluator. Auditors have 
found that in trying to prepare for an educational program audit, districts were faced 
with a new and difficult task. That was to devise an auditable evaluation design. To meet 
this need, the Office of the Los Angeles County Superintendent of Schools trained and 
assigned personnel to assist districts to write evaluation designs which would include 
measurable objectives, both product and process, a management plan, a time frame, 
identification of measuring instruments, and a reporting format. At this point, the Auditor 
undertakes a critique of the evaluation design. 

The Auditor must begin the audit with an analysis of the design features necessary 
to insure quality control. The foUowing questions serve to define the criteria for judging 
design quality: 

1. Are the decision situations to be served adequately defined? 

2. Are the research questions of interest clearly delineated? 

3. Do the research questions adhere to the deJsion situations to be served? 

4. Are the data to be collected adequately specified and do they match the research 
questions of interest? 

5. Are all questions investigated? 

6. Are the relevant populations and sampling procedures for data collection 
described? 

7. Are the procedures valid? 
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8. Are the instruments for data collection adequately described? 

9. Are they valid for the questions being investigated? 

10. Are schedules specified for information collection? 

11. Are formats and means for coding, organizing, storing, and retrieving data 
specified? 

12. Are data analysis procedures specified? 

13. Are the procedures specified appropriate for the situation? 

14. Is a schedule specified for reporting relevant information to specified 
decision-makers? 

15. Is the evaluation schedule presented? 

16. Is the evaluation schedule given staff and resource availability? 

17. Is the evaluation schedule realistic? 

18. Is the evaluation design likely to provide useful (i.e., valid, reliable, objective) 
information? 

19. Are there provisions made for process evaluation, that is, for observing the project 
in operation to determine whether or not it is functioning according to 
specifications? 

20. Is the evaluation budget adequate to carry out the proposed evaluation? 

It is essential to emphasize that this list h not all inclusive, nor is it arranged in 
priority sequence. It is assumed that variations of these criteria would be developed for 
use in terms of the educational project being audited. 

It is the function of the project evaluation team to select evaluation instruments 
which will measure the behavioral outcomes previously specified as objectives of the project 
under consideration. The EPA must verify that the instruments chosen by the project 
evaluation team actually have the capability, if administered correctly, to provide the kind 
of data relevant to the decisions to be made. If, in the opinion of the EPA, the proposed 
evaluation instruments do not meet the criteria of relevance, then he must suggest 
alternative instruments. 
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in order to perform this task effectively, the EPA must be: 

1. Totally familiar with the objectives of the project. 

2. Thoroughly conversant v^ith the objectives of the evaluation. 

3. Able to define the nature of the decisions to be served by the evaluation data. 

4. Familiar with reliable sources to validate his opinions of test instruments. 

5. Knowledgeable that the instrumentation will effectively measure behavioral 
outcomes. 

The evaluation phase of the educational audit can best be represented by a flow 
chart (see Figure 1) which moves from left to right beginning with the goal (Step 1), 
which is then spelled out in terms of objectives (Step 2). Next (Step 3), specific procedures 
and techniques to achieve these objectives are designed and described. In the flow chart 
the observable behaviors associated with each objective and equivalent specificity regarding 
the procedures and techniques to be used to achieve these objectives would normally be 
stated. For brevity, they are omitted here. 

Formative evaluation in the instructional area (Step 4) is intended to provide on-going 
feedback to the teacher to determine how well students have mastered various elements 
in a particular instructional program so that decisions can be made on how instruction 
should best proceed. It attempts to answer such questions as, "How are things going?" 
"What seems to be working or not working?" "Are some changes or additions needed?" 
Formative evaluation, as the term implies, is intended to help improve the instructional 
program. Summative evaluation (Step 5) is terminal in nature. Essentially, it aims to answer 
one question: Were the objectives achieved? In other words, at the end of instruction, 
where do the pupils stand in terms of the objectives initially stated in Step 2? 

In Figure 1 , which takes an arbitrary instructional goal, the basic format for graphically 
illustrating the evaluation process has general application, whether one is concerned with 
a management plan, an instructional program, or an educational support program. 
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Critique of the evaluation design is a key activity which is conducted by the Auditor 
to ensure the auditability of a progiam. The Auditor's task is to verify that project 
objectives are stated in such a maimer that they can be evaluated. To make sure that 
the recommendations are clearly understood, both a conference and a written critique 
are recommended. The foUowing was listed as an objective in the evaluation design 
submitted to the Auditor by one school district. 

"Has the ability to find information." A clearly stated performance objective should 
specify WHO, WHAT, WHEN and HOW MUCH. After the Auditor's critique, the district 
revised the objective as follows: 

"By the end of the school year, 90% of the ninth graders participating in the program 
who took both pre and post tests will demonstrate greater ability (significant at the 
.05 level) to find selected information than a comparison group." 

Another important aspect of the critique is a review of the appropriateness of the 
measuring instruments. In another district, the District Evaluator administered a pre test 
to establish baseline data. The Auditor found that even before participation in the program, 
50% of the students already scored above the 99th percentile on the proposed test. It 
was the Auditor's responsibility to advise that this was not an appropriate instrument, 
and to recommend others which would encompass the relevant parameters. 

The Auditor should also scrutinize tests for appropriateness of content. In one case, 
an Evaluator was planning to measure Reading Comprehension with the Jastak Wide Range 
Achievement Test, a test which includes only word recognition as a measure of Reading. 
Here again, the Auditor's responsibility was to advise that the instrument was not designed 
to measure the stated objective, and to recommend alternative instruments. 



ERLC 



« 



- 13- 

The Audit Contract 

The following contract form can serve in helping the auditor to develop an audit 
contract with a local school district. Following is the forrn in use by the Office of the 
Los Angeles County Superintendent of Schools in auditing programs in school districts 
in federal projects. The auditor and the LEA are cautioned, however, that any contract 
between them should be specific to the project to be audited. This form should be modified 
to conform to the requirements of both the auditor and the LEA. 

OFFICE OF THE LOS ANGELES COUNTY SUPERINTENDENT OF SCHOOLS 

AGREEMENT 
FOR 

EDUCATIONAL PROGRAM AUDIT 

The OFFICE OF THE LOS ANGELES COUNTY SUPERINTENDENT OF SCHOOLS, 
hereinafter also referred to as County, and 

, hereinafter also referred to as District, 

mutually agree as follows: 

WHEREAS, the County is involved in providing education program audit services, 
hereinafter also referred to as Audit; and 

WHEREAS, the District is in need of education program audit services of the 

The County shall perform audit services for said 

Program, hereinafter also referred to as Project as follows: 

1. SERVICES AND PRODUCTS TO BE PROVIDED BY COUNTY 

a. Provide Audit Reports 

The County will provide three audit reports in accordance with Paragragh 5 
and 6 hereof. 

b. Conduct On-Site Visits 

The County will conduct on-site visits in order to become familiar with the 
Project^s major components, to conduct interviews, to observe project 
administration, to conduct spot check of the evaluator*s procedures, and to 
review the operational aspects of the various project components. 
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c. Periodic Progress Report Meetings 

The County will hold periodic progress report meetings with the District 
Superintendent, the Project Director, and the Evaluator, as required, in order 
to assure open communication and to discuss recommendations. 

PROJECT PERSONNEL 

a. County personnel who will take part in the conduct of the Audit are described 
in the resumes. Said resumes are attached to this agreement and made a part 
of the agreement as though written into the body of the agreement. Any changes 
in assigned staff will be contingent upon approval of the Project Director and 
the appropriate County representative. The County's tentative plan for utilization 
of assigned personnel will be: 



FACILITIES AND SERVICES TO BE PROVIDED BY DISTRICT 

a. The District will permit the use of office and/or conference room space with 
appropriate writing surfaces to facilitate completion of any paper work while 
conducting on-site visitation. 

b. The District will provide the following documents during the first week of the 
^Audit agreement period: 

(1) Federal Regulations 

(2) Guidelines and Policy Statements of the District 

(3) Complete Project proposal 

(4) Pertinent correspondence between the District and the funding agency 

(5) Copy vof contract between the District and any other technical assistance 
source affecting the Project and the County's audit 

(6) All other reports and documents developed during the Project will also be 
made available. 

AUDIT PLAN 

Specific audit sampling techniques will be utilized to complete the Audit. These 
techniques and related activities are described in the Audit Plan attached hereto and 
made part of this agreement as though written in detail into the body of the 
agreement. 



1. 
2. 
3. 



(auditor) 
(auditor) 
(auditor) 



SCHEDULING SPECIFICATIONS 



a. The County shall complete major activities of the Audit, including the completion 
and delivery of audit reports, according to the following schedule. 

(1) Critique Evaluation Design 

(2) Complete Audit Plan 

(3) Complete Interim Audit Report #1 

(4) Complete Final Audit Report Ten (10) workdays 

after receipt of Final 
Evaluation Report 

b. Unless otherwise mutually agreed, any delays by the District in furnishing 
required information to the County will authorize the County to make equivalent 
adjustments in the Audit schedule. 

c. The conduct of all on-site audit activities will be approved by Project Director 
prior to actually carrying out the activities. The County will spend at least three 
(3) days on site to observe and become familiar with the major Project 
components and related activities. The County will request authority to conduct 
on-site visits five working days prior to the desired date, except for one (1) 
unannounced site visit. Approval or rejection of such requests must be made 
at least two days prior to the scheduled on-site visit. 

AUDIT REPORTS 

a. All preliminary drafts and final reports of audit activities and findings will be 
presented directly to the District Superintendent. Any subsequent release to other 
individuals, firms, or agencies shall be approved by the District Superintendent. 

b. Five (5) copies of each audit report will be delivered to the District 
Superintendent. 

c. The County will hold periodic progress report meetings with the Project Director 
and the Evaluator in accordance with Paragraph 5 hereof. 

d. Audit reports will include, but are not limited to, the following contents: 

(1) Introductory and general comments concerning the quality of the project 
evaluation and the comparative findings of the project evaluation and the 
Audit. 

(2) Detailed critique of the comprehensive evaluation conducted, by 
component, based on an assessment of the instruments used, data collection, 
data analysis, and data analysis presentation procedures. 
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(3) Description of the County's on-site visit findings and their correlation with 
the Project Evaluator's data and reports on a component-by-component 
basis, summary of consistencies and discrepancies, and interpretation of the 
discrepancies. 

(4) Recommendations for revisions in the evaluation design, including a 
rationale for each recommendation. Since the Co^mty's objectivity can be 
retained only if the selection of a specific corrective action is a local 
decision, the County will provide recommendations posing alternative 
actions or possible sources of assistance to the Project in correcting the 
deficiency. 

(5) Confirmation or questioning of the need for program modifications wliich 
have been proposed as a result of Project evaluation. 

e. The County will be responsible for the preparation and certification of all audit 
reports provided to the District. 

7. ASSURANCE OF CONFIDENTIALITY 



a. Any need by the County to have access to specific documents or persons will 
be made known to the Project Director. Such requests are expected to include 
only those items outlined in the OE, "Outline of Education Program Auditing 
Procedures," and only to be from those persons directly involved in the Project. 

All information and findings related to the Audit will be held in strictest 
confidence by the County. Any nonconfidential publicity, journal articles, or 
other printed matter of a dissemination nature which may be developed during 
the contract period will be presented to the District Superintendent or his 
designated representative for concurrence and approval prior to publication or 
release by the County. 

8. PAYMENT SCHEDULE 



a. Upon completion and delivery of the required audit reports, the District shall 



pay to the County the sum of . within ten (10) days 

of receipt of an itemized invoice. Payment to be made as follows: 

Critique Report $ 

Interim Report #1 S 

Interim Report #2 $ 

Final Report 10 days after receipt $ 

of Evaluation 



b. For each major report component required in accordance with Paragraph 6(D), 

which is not included in audit report, % of the payment due 

upon the submission of that report shall be withheld until the report is 
completed. 
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c. For each week that an audit report is overdue in submission, % 

shall be deducted from the total payment amouut. 

9. INDEPENDENT CONTRACTOR 

While performing services as contemplated by this agreement, County is an 
independent Contractor and not an officer, agent or employee of the District. 



10. HOLD HARMLESS 




a. County agrees to maintain adequate workmen's compensation and liability 
insurance and to hold District harmless and to indemnify the District from every 
claim, demand, or liability which may be made by reason of: 

(1) Any injury to a person or property sustained by County or by any person, 
firm, or corporation employed directly or indirectly by c County upon 
or in connection with the work called for in this agreement, however, 
caused; and 

(2) Any injury to a person or property sustained by any person, firm or 
corporation caused by any act, neglect, default, or omission of the District, 
upon or in connection with the work covered by this agreement, whether 
the said injury or damage occurs upon or adjacent to the work; and 

(3) County, at its own risk shall defend any and all actions, suits or other 
proceedings, that may be brought or instituted against the District on any 
such claim, demand or legal proceedings or result thereof. 

11. STANDARD OF WORK 

The Contractor agrees that all work will be performed in accordance with the highest 
professional standards. 

12. DURATION OF AGREEMENT 

This agreement shall be in full force and effect upon execution. This agreement shall 
be subject to amendment and/or termination by mutual consent of the parties, in 
writing, in which event both parties shall be discharged from all obligations hereunder, 
except those obligations for reimbursement as may be accrued but unpaid on the 
date of expiration. 

This Agreement shall be in full force and effect through . 



OFFICE OF THE LOS ANGELES COUNTY 

SUPERINTENDENT OF SCHOOLS SCHOOL DISTRICT 

By By 

Contractual Relations Officer 

Title 

Date Date 
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The Au Jit Plan 

The Audit Plan is an essential part of the Audit Process. It is a phnning and operational 
control document for the EPA and a quality and ma.iagement control document for the 
local project director. The plan indicates the techniques, schedules, processes, and 
procedures which the EPA will use in judging the adequacy of the evaluation process 
and in verifying the evaluation findings. 

At this point in the sequence of events, the Auditor identifies those aspects of the 
evaluation plan upon which he will concentrate his reviews, on-site activities, and data 
analyses. 

There are certain elements which should appear in any audit plan. They include 1 ) a 
notation of the types of activities to be performed, 2) when they are to be carried out, 
3) how many work days are required for their completion, and 4) when specific documents 
or reports are to be provided. 

The Auditor develops a list of his planned activities in the sequence in which they 
will occur. In Example A which follows, he then transfers them to blocks, connects them 
in order of occurrence, and plots them against a time line. £)ecision or conference points 
and report availability dates are also displayed. Audit Plan A displayed below incorporates 
each activity into the responsibility areas of data collection, data analysis, and data 
reporting. 

Audit Plan B displays the project objectives and the Project Personnel Responsibilities 
and Auditor Responsibilities related to each objective. 

The advantage of displaying the audit plan is that all major activities are displayed 
in relationship to each other and also to the overall project time line. It also requires 
the auditor to be highly specific as to when he is going to perform a task and how long 
he is going to take to complete it. Numbering of the major task objectives, a breakdown 
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of the task, and a detailed explanation of the planned work activitks is recommended 
as a companion piece for this display. 

The audit plan is one of the most useful tools for facilitating the audit process. 
By visually displaying the planned audit activities in relation to project and evaluation 
activities, the purpose of the audit efTort is clearly understood. The audit plan should 
identify who is responsible for each phase of the audit process, where each activity* will 
take place, and the techniques that are to be utilized. So drawn, this plan is valuable 
to the Auditor in scheduling workload and to all members of project management who 
will be dealing with the Auditor. 
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CHAPTER III 
The Interim Audit Report 



The interim audit reports should include the following: 

1. Verification of evaluation procedures and data collection 

2. Observation of testing 

3. On-site visits 

4. Spot-checks of materials and classes 

5. Notation of discrepancies 

6. Auditor's recommendations including alternative strategies and supporting 
rationale for revisions to the evaluation plan 

7. Reference to Checklist of Systematic Instruction - Preferred Practice^ 

The interim audit reports take two forms: 1) a formal interim report based on the 
progress of the program evaluation presented midway through the project year, and 
2) informal interim reports which should be submitted periodically and may be a simple 
letter or memo. 

Process-Formative Evaluation * 

The on-going phase of the audit process represents a highly significant and iniportant 
aspect of the formative-summative approach. Not only does this introduce the realistic and 
essential elements for self-correction, but it also maximizes the involvement of the 
educational staff in self-evaluation. Lastly, it provides, through the expertise and training of 
the EPA, critical judgments in regard to degree of progress, logistical analysis, and ultimate 
expectations. In determining whether the original basic objectives were well defined, 
realistic, and clearly understood, the following questions need to be considered: 

1. Are conditions and procedures to achieve objectives reasonable? 

2. Are conditions and procedures to achieve objectives practical? 

3. Are feedback processes appropriate? 

4. Are feedback processes broad based? 




'See Checklist of Systematic Instniction - Preferred Practice, pp. 23 and 24. 
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5. Are proper instrumentation techniques provided? 

6. Are constraints and parameters well defined? 

7. Are adequate resources available for effective implementation? 

8. Are effective inservice programs being conducted? 

9. Are sufficient opportunities provided for audit process? 

10. Are feedback channels working effectively to provide for substantive 
information? 

11. Are adjustments and modifications occurring as a result of feedback? 

12. Are evaluative instruments providing significant data? 

The culmination of the Auditor's on-site visit is an Interim Audit Report. This report 
should include those reporting requirements listed in paragraph 6d of the Audit Contract. 
The Interim Audit Report is an official report submitted directly to the Superintendent 
by the Auditor. As a matter of procedure^ however, this report is initially sent to the 
Project EHrector in time to allow him to develop rebuttals to any points with which he 
disagrees, or to develop plans to implement any corrections he agrees are needed. An 
example of an Interim Audit Report is shown on the following pages. 
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RICHARD M, CLOWES. Superintendent 
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December 12, 1972 



Mr. John Doe 
Superintendent 
Union High School 
Los Angeles County 

Dear Mr. Doe: 

In accordance with paragraph six of the Audit Contract, relating to the ESEA Title I 
Program, the following Interim Audit Report is hereby submitted. 

This report is based upon data generated as a result of the evaluation critique on 
November 3, 1972, and the on-site visit of the Auditor to A and B High Schools on 
December 6, 1972. 

1. On November 3, 1972, the Auditor met with the Project Director, Project 
Evaluator, and representatives of the State Department of Education to clarify 
the statements of objective^ and the evaluation procedures. The State Department 
representatives agreed with the proposed changes. No comparative findings of 
the Project Evaluation and the Project Audit are included in this report since 
none were planned to occur this early in the project. 

2. The Auditor concurs with the evaluation instruments selected for the project 
and with the proposed data collection, data analysis, and data reporting 
procedures described in the evaluation design. 

3. This paragraph reports data generated during the on-site visit of the Auditor 
on December 6, 1972. The evaluation design does not envision a Project 
Evaluator's report at this stage of the project, and no study of the correlation 
with the Auditor's findings was planned at this stage. The Auditor sampled the 
records of ninety-one students at B High School and fifteen students at A. At 
B High all students in the sample were scheduled into a laboratory period as 
provided in the project proposal. At A, all but one of the students in the sample 
were scheduled into the laboratory period. The exception was a student who 
had arrived the day prior to the Auditor's visit, according to the Principal. The 
nurses records of the same sample at both schools were examined to ascertain 
whether they had been screened with audiometer and telebinocular. At B High, 
seventeen with the telebinocular. Fifteeti of this group of students had had both 
tests; two received only the telebinocular and one received only the audiometer 
screening. At A, nine students in the sample had received the screening on both 
the audiometer and telebinocular. 
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Mr. John Doe 



December 12, 1972 



The project proposal provides that all project participants would be scheduled for 
this screening in September-October, 1972. Based on the Auditor's sample of approximately 
one-third of the project participants, only twenty-five percent of the students had been 
screened at the date of the Auditor's on-site visit. It should be noted that the project 
budget makes no provision for Health Services expenditures. The same nurse who provides 
district wide health services also i^ assigned to conduct the required health screening for 
project participants. The B High Principal observied that a temporary person had to be 
hired from district funds to supply needed health services. 

The Auditor examined the on-going evaluation records of Project Teachers and Project 
Aides. The evaluation records of all Project Teachers indicated that only two of the five 
Project Teachers had been evaluated by the date of the Auditor^s on-site visit. Records 
of the Aides indicated that all had been evaluated to date. However, the evaluation was 
made by the classroom teacher and not by the supervising Principal or Vice Principal 
as stated in the proposal, and was not related to individualizing of instruction. 

The Auditor examined and rescored a random sample of the approximately 350 
reading and math pretests. It was found that 41.7% of the reading tests at A were 
inaccurately scored; 5 tests in 12. At B High School, 15.1% of the reading tests were 
inaccurately scored; 13 tests in 86. The math tests at A were inaccurately scored in 15.4% 
of the cases; 2 in 12. At B High School the math tests were inaccurately scored in 32.6% 
of the cases; 28 in 86. Overall, the reading tests were 18.4% inaccurately scored; the 
math tests 30.3%. 

In the reading test, sixteen of the inaccurately scored tests penalized the student 
from 1 to 9 raw scores. The remainder of the inaccurately scored tests added one or 
two raw score points to the students scores. 

It should be noted that the magnitude of the scoring error for an individual student 
was sufficiently small that there was no instance in which a student's placement in the 
quartile was altered. 
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Mr. John Doe 



December 12, 1972 



TABLE I 



Number of Students in Ql 82 83% 

2 9 9% 

3 6 6% 

4 2 2% 



Required Number of 70 70% 

Students to attain 15 15% 

reading objective 10 15% 

5 5% 



Table I indicates that only 12 students in the first quartile in the Auditor's sample 
population must move up to the second quartile to attain the reading objective. It is 
not necessary that any of the students in Q2, Q3, or Q4 improve in order for the project 
to attain the stated objective in reading. 

An examination of the math scores of the population sampled by the Auditor 
indicated the following: 



TABLE II 



Number of Students in Ql 49 Required Number of 70 70% 

2 25 Students to attain 15 15% 

3 12 math objective 10 10% 

4 14 5 5% 

Table II indicates that no student need move up to a higher quartile to meet the 
stated math objective. 

4. Recommendations: 



A. Reexamine the project requirement that all project participants be given 
audiometer and telebinocular screening. 

1. Reexamine the nurses' practice of giving multiple screening to students 
who initially do not pass the test. 

2. Examine the feasibility of supplementing the present nursing senrices 
from district or project funds. 

B. Apprise the Principal/Vice Principal of the Project requirement that Project 
Teachers and Aides have on-going evaluation reports related to 
individualizing of instruction in language development and math made by 
the Principal/Vice Principal. 

C. Examine the sensitivity settings of the optical sca>mer test scoring device 
to increase the accuracy of test scores. 
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Check the machine score accuracy by running the Right-Wrong answer 
key through the scanner after each twenty-five to fifty answer sheets. 



Mr. John Doe 



December 12, 1972 



2. Recheck all hand scored answer sheets. 

D. Examine the validity of the stated objectives in Language Development and 
Mathematics since no improvement is required in math and a minimal 
number need to improve in reading to attain the objectives. The present 
objective requires that approximately 14% of the projected 245 students 
in Ql need to move to a higher quartile to attain the objective. 

1. Examine the per pupil cost of the gain of the projected 36 students 
in the project who are expected to move to a higher quartile. 

5. The Auditor recommends no program modification. 

Sincerely yours. 



Auditor 
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CHAPTER IV 
The Final Audit Report 



The final year-end audit report should present comments on audit activities which 
have been conducted since the last interim report as well as a summary of the audit 
finding and recommendations for the entire contract period. 

This chapter discusses and illustrates the type of information which should be included 
in the final audit report. The five content areas suggested by the United States Office 
of Education for inclusion in the report are: 

1. Introductory and general comments 

2. Detailed critique of the product and process evaluation 

3. Description of the Auditor's on-site visit findings 

4. Recommendations for revisions in the evaluation design 

5. Confirmation or questioning of the need for program modifications 

1. Introductory and general comments concerning the quality of the project evaluation 
and the comparative findings of the project evaluation and audit. 

This area describes the general status of the project evaluation and includes the purpose 
of the final audit report. It states the activities accomplished by the Auditor such as 
critiquing the evaluation design, interviewing the project staff, and conducting oa-site 
visitations and observations and lists the overall findings which resulted from thes ''ivities. 
An example of such findings might be: 

"The findings of the Auditor agree with those reported in the final evaluation 
report.** 

Certain program operational processes which were observed by the Auditor and found 
to be correctly reported may be verified by listing the processes, such as: 

"The personnel reported in +he original pv^posal were observed on the job in 
actual project operation. The number of project ^'eachers and specialists were in 
agreement with district records.'" 
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Detailed critique of the product and process evaluation conducted for operation and 
management in each component, based on an assessment of the instruments used, data 
collection procedures, data analysis techniques, and data analysis presentation. 

This area discusses each section of the final evaluation report, such as Introduction, 
Budget, Summary and Recommendations, Evaluation of Project Objectives, and Appendix. 
The critique and verification are based on United States Office of Education guidelines 
specific to the project. Typical comments (kept brief for the purpose of providing examples) 
which might originate from these sections are: 

1. "There were no unusual budgetary changes and the funds were used as budgeted." 

2. "The evaluation design did not incorporate an analysis of the gathered data." 

3. "The Reading Component was generally much better organized than tne other 
three components." 

4. "With the exception of published tests, the appendix contains the instruments 
used for the project's data collection." 

5. "Although objective number ten was not met, it can be noted that 80% of the 
students in the project increased in positive behavior." 

6. "Objective number seven does not include the instrument to be used to measure 
the desired performance." 

7. "The summary and recommendation sections contain an excellent summary of 
the attainment of performance and process objectives." 

Description of the Auditor's on-site visit findings and their correlation with the 
evaluator's data and reports on a component-by-component basis, summary of consistencies 
and discrepancies, and interpretation of the discrepancies. 

All the on-site visits which were conducted should be listed by date, activity 
performed, and member of the audit team making the visit. Specific activities performed 
may include reviewing program records, reviewing any audit reports with the administration, 
meeting and interviewing with the program staff, and observing testing procedures and 
administration. Example findings which might evolve from these on-site visits include: 
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1. "The Auditor reviewed the records of all project students to see that all tests 
were included as stated in the objectives. There were no discrepancies in the 
data." 

2. "Discussion with the Project Director focused on modifications for next year. 
He stated there would be a move to implement self-contained classrooms for 
all grade levels." 

3. "Since the Auditor did not review the actual testing situations, this audit can 
only verify that the data were collected and reported for each student." 

4. "The Auditor interviewed teachers to obtain general comments on the program. 
One teacher commented that the Social Studies books were not used this year 
since they did not coordinate with the objective." 

If there are any problems (i.e., in reference to staff personnel or program support), 
these may Jbe described and suggested solutions may be made by the Auditor. 

4. Recommendations for revisions in the evaluation design , including a rationale for each 
recommen dation. 

Since the Auditor's objectivity can be retained only if the selection of a specific 
corrective action is a local decision, he should provide general rather than specific 
recommendations, posing several alternative actions or possible sources or assistance to 
the LEA in correcting the deficiency. 

These may be recommendations for improvement as a result of critiquing the final 
evaluation report, or they may be recommendations that were in the pre-audit critique 
or interim reports which the Auditor would like to reemphasize. Example recommendations 

are: 

1. "The project should continue to request financial support to develop the remedial 
program. This is a well-organized project and the auditor feels it is entering 
a phase of refinement." 
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2. "Since there are so many recommendations incluJed in the fmal evaluation 
report, it might be wise to place priorities on these which are felt to be most 
vital to the overall success of the project/* 

3. "Performance and process objectives should be developed for each component 
of the evaluation. Data must be related to specific objectives if the results hj 
to be interpreted so as to measure attainment of the objectives and thereby 
measuring the success of the program." 

5. Confirmation or questioning of the need for program modification s which have been 
proposed as a result of project evaluation. 

This area includes Auditor comments on the recommendations of tne evaluation 
reports. These modifications may also have been confirmed in the interim audit report. 
These changes are not always m^jor changes in the evaluation design, but they do reflect 
the opinions of the Auditor which he feels will refine and improve the project. They should 
not be in conflict with district guidelines. 

The development and delivery of the audit report is not all that is required of the 
Auditor. The Auditor should review and discuss the contents of all reports in a joint 
conference with the Project Director and the Evaluator. There are several reasons for this 
requirement. It provides an opportunity for project and evaluation staff to point out any 
discrepancies or misunderstandings which the Auditor may have inadvertently incorporated 
into the rationale upon which his recommendations are based. It also makes the Auditor 
available to project and evaluation staff for discussion of specific audit recommendations 
and findings and for clarification of any items which may not be fully understood. 

The final report is the Auditor's certification of the evaluation design or his criticism 
of any facet of this design which he feels is not adequate to the task of properly evaluating 
project process, product, and management* 
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SUMMARY 

The Educational Program Auditor needs to be aware of the impact of an outsider 
looking in on a system. For some personnel, the Auditor's activities may be construed; 
as a threat ind generate anxiety. Both the Auditor and school administrators should clarify 
to district personnel that the Auditor's function is to emphasize feedback rather than 
inspection. The \uditor s attention is directed to the congruence between the district's 
evaluation design and its implementation of the program, not to criticism or evaluation 
of individuals. Interactions with personnel at every level should be conducted in the spirit 
of redesign for quality assurance. All communications, including audit reports, should be 
clear, explicit, and understandable to the intended reader or receiver. 

As Lessinger^ has said, the application of the educational program audit as a feedback 
loop to educators will guarantee the acquisition of basic skills by all of our children through 
the concept of accountability. Such a process can revitalize the public schools, save society 
the long-term cost of allowing its schools to define millions of children as "failures," 
and assure that as taxpayers we get our dollar's worth. 



Q ^Lessinger, Leon. Op. cit., pp. 12 and 19. 



